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The findings from this review do not reflect the full body of research evidence 
on the Kalamazoo Promise Scholarship program. 


What is this study about? 

The study examined the impacts of the Kalamazoo 
Promise Scholarship program on academic and 
behavioral outcomes of students in grades 9-12 
in Kalamazoo Public Schools (KPS). The program 
provides KPS students with scholarships to attend 
any community college or public university in Michi- 
gan. Scholarships are promised to those meeting 
residency requirements, so it is possible for students 
to know they will be eligible many years in advance 
and they may adjust their behavior accordingly. 

Study authors compared the academic and behavioral 
outcomes in high school before and after the Promise 
Scholarship program was introduced. The interven- 
tion group included students who were eligible to 
receive the scholarship, and the comparison group 
included students who could not receive the scholar- 
ship (because they had not attended a district school 
for long enough to qualify). The analysis was based on 
approximately 14,500 student-year observations from 
the 2003-04 to 2007-08 school years. 

Student outcomes that were included in the study 
were: student grade point averages, whether stu- 
dents earned course credits, the number of course 
credits earned, incidence of and number of days 
spent in suspension, and incidence of and number of 
days spent in in-school detention. 

What did the study find? 

None of the analyses presented in this study meet 
WWC group design standards, and therefore, the 
study findings are not presented in this WWC report. 


Features of the Kalamazoo Promise 
Scholarship Program 


The Kalamazoo Promise Scholarship program offers 
college scholarships to graduating high school 
students in the KPS district. 

The scholarship covers up to 100% of tuition 
and fees for attending any community college or 
public university in the state of Michigan for up 
to 4 years. The actual percentage of tuition and 
fees covered varies depending on how long the 
student attended school in the district. Students 
attending since kindergarten receive the full 100%. 
Students attending since ninth grade receive a 
scholarship covering 65%. Students who enter KPS 
in tenth grade or later are not eligible to receive the 
scholarship. 

Other than the KPS school enrollment requirements, 
and the requirement that a student is admitted 
to and attends a community college or public 
university in Michigan, there are no GPA or financial 
need requirements to qualify. Thus, it is possible for 
students to know their eligibility for the scholarship 
for many years. 


WWC Rating 


The research described in this 
report does not meet WWC 
group design standards 

This study uses a quasi-experimental design in 
which baseline equivalence of the groups cannot be 
demonstrated. 3 
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Endnotes 

1 Bartik, T. J., & Lachowska, M. (2013). The short-term effects of the Kalamazoo Promise Scholarship on student outcomes. In S. W. 
Polachek & K. Tatsiramos (Eds.), New analysis of worker well-being research in labor economics (Vol. 38, pp. 37-76). Bingley, United 
Kingdom: Emerald Group Publishing Limited. 

2 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC design standards. The review reports the WWC’s assessment of whether 
the study meets WWC design standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the single study review protocol, version 2.0. A quick review of this study was released 
on May 14, 2014, and this report is the follow-up review that replaces that initial assessment. 

3 To assess baseline equivalence of the students in the sample, the WWC requires information on baseline demographics and achieve- 
ment separately for the intervention and comparison groups. However, this study uses district data from Kalamazoo that includes stu- 
dents who moved into the district after the intervention was implemented. The data for these students influence the impact estimates, 
but baseline characteristics are not available since they were not in the sample until they moved into the district. As a result, there are 
some students included in the analytic sample for whom it is not possible to demonstrate equivalence prior to the intervention. There- 
fore, baseline equivalence for this design cannot be demonstrated. 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 

A substantively important finding is one that has an effect size of .25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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